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ABSTRACT 

Problems associated with combining observations taken 
from intaqt classrooms were examined. Clarification of basic entities 
to be studied was a critical step in research planning. Classroom 
6fcservers usually, study instructional arrangements and other 
classroom phenomena. The author used activity structure and i^s 
component activity segments as. the focus for study* An activity^ 
segment was^ defined as a unique time block in a lesson which occur ed 
in a f ixed^physical setting. Segments have instructional forms 
^ designed for the accomplishment of specific tasks. Daily activities 
ih'^a classroom conform with the general, type of curricular approach 
and goals. Activity structures are produced to accomplish certain 
-goals. When observational data, i.s characterized by distributions 
which are bimodal or Have many zeroes or non-ocourences, summary. with 
means land standard deviations may result *in erroneous conclusions. . 
-The author observed ^20 fifth-grade math classes from diverse school 
districts. He determined that combining data within two clusters, ^ 
highly individualized and whole class programs, provided more 
meaningful and interpretable results. Researchers* and writers need to 
include background descriptive information about the programs and 
contexts which they have studied to facilitate/sorting of data. 
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Combining Evidence Between and Within. Educational Research Studies 
Susan S. Stodolsky', University * of Chicagp - 

The use of direct observation for studying classroom prooipfs. and 

learning outcomes has increased noticeably 'in the last ^ecad'e. ^studies 

of schooling as JLt actually occurs in classrooms 'are being c?mciucted to 

fill our need for ^timely descriptive information and-%>\jieljjf operationally 

definp actual -educational treatments: However, observational studies of 

classroom processes require new methodological skills L 0f& strategies. 

at? 1 * 

One particular aspect of observatipna^ ■•methodology will be 1ny focus 
in this paper. I will address problems asspci^ted* with^ combining ©bssrva*- 
tions taken from intact classrooms. Iri" considering this topic J will^draw 

parallels between combining clata within 6 3, given jjobservati'onal study and 

' a- \ K - >* ' \ * y * / ■ ' * 

that of combining dafcfca from a number of di^feMnt studies as is done in 

V ' l . 

quantitative research synthesis. • / ... 

* v ; if : 

•Direct observaftionVf classroom^ lis 4xecuted with a wide variety of 



specific observational methods. 



tjple, teacher-student interactions 



have often been obsejrved using suck instruments as the Flanders Interaction _ 

' ^ • • • , 

Analysis Category system (Flanders, 1960) ,or afehe Brophy and pood system (1970) 
Another approach isj th£ use of narrative recording or specimen records as in 



studies of third-grkde classrooi 



k Gump (1967), videotaping of whol£ 



lessons analyzed* for critical' i^bidents anci judgements of tocher character- 
istics (Kounin, 1970)./ and o€h,erMdaptations of opep recording methods 
such as I have used in researM^'on ' fifth-grade math and social studies 

t\ 1981).'- M: ■ , ! 



classes .(Stodolsky 



* Paper presented at^h£ annual meeting of the American Educational 
Research Association, Nei^tfoxfk, March, 1982. 
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A variety of -decisions must be made when conducting non-experimental 



field studies of cl^ssrtom processes. Obviously, the study must be framed 
and operationalized with clear questions and hypotheses in -mind and an * ' 
understanding of ,the kinds of generalizations which* are desired. A critical 
step in the research planning process is clarification of the nature of the 
phenomena to be studied. ! As -Dunkin' arid- BiddLe 0-974) ,have pointed out,'* 
even, the mq^st .obvious spunding terms* can be extremely ^ambiguous when opera- 



v * fri^nalizaGton^ds r^quired^' 'For/'exatfipl^, t*he terms M, le 
-'/and even "student'^ may prove .jflifficu^t^tg define in ce 



lesson, ,f "interaction" 
"to define in certain contexts. , " t 



°Curricular labels ^uch'^s opfen education or individualized instruction often H 

» ■ - > , 4 -.V ' . * » - V . • 

t * ^ , *** ' . ''*' 

coanote various and sometimes t conflicting meanings. Clarification of the, 
basic entities* to be studied deserves careful' attention because findirtCfs are • 
mo£e easily interpreted ^ if * the ^ phencSciena being* studied are clearly defined. 

• Onse such clarification of questions and -phenomena is achieved,* 
observational re^earchfers must decide on>some sampling plan. Decisions are 

* " m '* * 

made abc^it whatt *to observe,, how to observe, when to observe, and how long 

to "observe. These decisions atre made with an eye toward the desired generali- 

\ ' '?V. •; • "* J . \ 

zations but also wi t hi JjT practical constraints such as 'available resources 

.and teachers Y willingness to' admit outsiders %b their classrooms. 

y*. '.'•'« . * ' * ' ' ' 

* The clarification process involved in planning observational research 

• is somewhat parallel to ^initial work which must Be done' in effectively 

\ planning a quantitative research synthesis. The -parallel is strongest in 

\ , 

connection with the process of deciding what studieis to include in a* synthesis 
This deoision must be made by considering the construct equivalence of bojh 1 
treatm^M^an£ measures in diverge studies. I will return to thes£ parallels 
later in* this paper. ' , * 



.Once conducted, what kinds of results do observational studies produce? * 
Typical are findings which describe the^f requencies and/or rates of occurence 
of certain behaviors and other learning conditions. Flander's "law of 
two-thirds' 1 which asserts that 2/3 of the- time in cla^srdoms is spent'^in 
ta*ik and 2/3 of the talk is done *k}y the teacher Is an example. Another 

* example', from a recent British study of junior schools (Gal ton, guncgi, anQ 
Croll, 1981) is that 10 percent 1 of all 'activities observed Across classes 

' ■ ■ ' ' ""V' : " - - ' v ■ 

in a year's time ar£ cooperative grpup^tap^k. ^' In my^ research (S^cfolsky, 1 

1981| in fifth-grade math anc^ sbpial studies classes we found that group 

wori^in social studies 'occupies' 10.7 per cent of "student time across 19 
*' • • * * » 

different classes Observed 'for two weeks each. We also found , that individ- 

•* * 

ualized seat work in mathematics occupies 13.7 per cent of student time 

across 20 different math classes we observed "for- two weeks each. 

These statements about classroom phenomena and their occurence are % 
• » 

wmeant to inform us in some way -about the topical .experiences of teachers* 

* v ■ , r 

and students as well as the condi,ti<ans for. teaching and learning which are / 
present in our schooLs. Often an additional step is /taken in which such 
Variables^ are correlated with learning outcomes in osder to- construct a ' 
<5hain of connections between educational processes and .educational effects. 

The issue to be raised is the appropriateness of aggregating' data 
taken from a variety of intact classes, lessons, o£ educational environments. 

• L • 

What substantive and quantitative assumptions should be met; in ordor to 

summarize data 'across classes using standard descriptive statistics such* , h 

as means and standard deviations? How do we decide if the interpretation 

of f such data .will be valid? For example,' shall we assume that children in 

** • * 

4 % 

British junior ^schools do group work activities* 10% of -the time? 



1^ address this query first from 'a theoretical and' conceptual 

* - . • f , ' . 

perspective about tlje nature of instruction. The variables usqally 

studied by observers are aspects of instructional arrangements and other 
• » * 

- classroom, phenomena. Instructional arrangements are significantly and 

i 

fundamentally constrained by educational purpose and organization and ,are 

•*« • 
hi^hlyt interdependent phenomena, In my own work I view educational set- 
tings through the use of an ecological perspective, usipg the activity 

• strfcctvure-and its component . activity segments as the focus for -study. 
The concept of an activity segment is~derived from the idea of a behavior 
setting, a fundamental concept in ecological psychology pfeark^r, 1968). 
An activity segment is part of a classroom activity structure which has 
a particular instructional rormat, participants, materials, Behavioral 

.expectations and *goals, and space-time boundaries, A segment is defined 

as a unique time block in a Wesson and occurs ia a fixed physical setting. 

Segments can occur -singly as when the whole class is involved in the same • 

activity such *as a teacher-led recitation or simultaneously as when four 

grpups of children ar§ working on tasks . 

Knowing a particular activity segment exists does nbt permit one 

to predict all the molecular behavioral transactions in the classroom, 

but it does significantly limit the likelihood of seeing certain behaviors 

as opposed to others* Segments have instructional forms which are suited 

\ - * 

Pcomplishment of certain tasks. While the detailed connections 
orm'and tasks must be investigated in more depth, it is clear 
fies of contextual* *and pedagogical factors affect the configura- 
ehaviors and activities which occur ih a classroom' at any given 
r example, knowing that cSildren are participating iiv-a whole- 
itation will suggest a context in which teacher talk is likel^ * 



I * 

to predominate and child~Child interaction' is likely to be very infrequent. 

# Or; the other, hand .knowing^ tihat children are doing seatwork in a math class* 

"will mean that teacher-child , interactions are, likely * to be private and that 

«» * 

f ' * y 

children will? be writing as a primary activity. In sucfh cases the cognitive 
* lev§l of the activity is not fixed,' although prior experience and research * 

would indicate that certain levels of question-asking 'are more likely to 

» * ■ * ' fc ' 

•occur in recitations and certain types % of, problems are likely to be solved 

in *a seatwprk setting. ' ^ 

\ Beyond knowing th^r properties of a particular setting and the con- 
straints that 'it imposes, the fact that settings and segments are designed*, 
to accomplish certain 6 curricular purposes must also B be considered. The 
daily activities 3 i« .a classroom are ordered and cohere at many levels, 
one important one is in conformity with the general type of curricular 
approach and goals ♦ Activity structures are produced to accomplish certain 

* opals. For example, a teacher; attempting to 1 implement an open education * 
program is likely ?fco structure the day to enable children to make a variety 
of*. choices about the activities they will pursue ✓ An individualized irtath 
program will probably consist of seatwork sessions in which children work 

1 at their o,wn pace, possibly intersjpersed^witht tests and teacher conferences. 

In social studies, fcertain curricula are predicated on the use of peer work 

* \ 

<. groups. For examplev^ife MACOS curriculum contains many 'activities wh^oh 

are -to be carried out by a group, of children working cooperatively. 

With programatic constraints operating, it is clear that classrooms * 
• % 

' must look quite different from one another. In fact this is the- essence • 
of educational diversity. , t Therefore it is insufficient to think of tfce* 
Variability as connected to $r associated with some typical average. In ^ 
fadt it may often be necessary to recognize interrelated patterns which must 

* I . t > 

- be distinguished 6ne from the other* * 



» A * 

#> . * ' 

' /Our own data provide some convenient illustrations of* this point* . 
J ♦ I indicated earlier that we observed, twenty fifth-grade math classes from . 

V * . * 

» r • • ✓ \ \ 4 • • 

diverse school districts for two consecutive weeks. -We * found that individ- 

- • • * 

* * 

' ualized seatwork in which children work at their own pace and use testing 

«.'••.•."• ' " • ' « •. ' 

. and diagnostic devices accounted for 13.7 percent of student occtipancy 

• • 

■ ♦ * 

time across^ all, classes* and* occasions. Thus ojie might expect that walking 

into a math class art £he . fifth grade, the observer would see 'children • , 

* * " occupied 'with individualized seatwork on one of every eight occasions. 

However, an inspection of the" data would show that this summary statistic 

is. a very poor summary indeed. Actually llr of A) classes never used the 
• * . i 

individualized seatwork approach. Four classes had this arrangement less 

than f;ive percent of the time (basically on one occasion) while in the 

other five clashes it occupied 71%, 70%, 55%, 32% and 19% of student time. 

Essentially we have a three-way -Clustering on this variable: 1) The pro- 

gram .(individualized seatwork) is not .used at all. 2) The program' is used 

■ infrequently for a special purpose. 3) The program is the major curncular 

approach utilized. To adopt another type of language, these clu'sters would 
, • t j % <j - * 

, i *6eem to reflect at least three different curricular treatments. To the 

extent that these classes are implementing ^different curricular approaches 
and treatments, it would seem essential to both substantively ^nd statistic- 
ally seperate them for /purposes "Of analysis* It does not seem appropriate, , 

at this level of analysis, to think of , them as forming, some sort of continuum 

I 

* to be used for- descriptive or predictive purposes. They are coherent 

* educational entities which are operating .with different assumptions, pracf- 

* . tices, and possibly different goals I c 

A similar configuration can be, found with regard to the occurence' 

* 

of peer instrufctional Work groups in social studies. While we have an 

* - 

ERIC . *• ; ' \7 . 
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t 

.average occupancy time v of 10.7 percent in group work settings across 19- 
classes/ 9 classes are never observe^ using groupwork settings, 3 'classes 
use them occasionally, and seven use them a Jsubstantial amount suggesting 

that they are a dominant instructional form in regard" to the curriculum 

* * * 

being utilized. Similarly Gaiton, Simon and Croll (1981) recently found 

that groupwork in British junior- schools did not occur at all for approxi- 
mately 90 percent of the children, even though groupwork accounted for 
10 percent of all activities observed across classes. « 

Observational data is pften characterized by distributions which 
£re either bimodal «or which have many zeros or non-occurences. To sum- 
marize such d^J:a with means and standard deviations Qr to use such vari- 
ables as predictors is perhaps* to assume too much (or the wrong things) * 
about our knowledge of the phenomena. In particular , when variables are 
distributed in ways we have just described for groupwork or individualized 
seatwork, displaying average occurences or correlating their occurence" 
with outcome variables suggest that we have a -variable whose sheer frequency 
will show an orderly relation to other variables. But iit these cases and * 
many other observational data contexts low or no frequency of Occurence 
on a given variable means that an entirely different educational program 
is in operation which must be evaluated a^s such. 

It seem? much more important and appropriate to attempt tc; identify 

* *> . 

treatments which are in fact similar on substantive grounds in order to 

better combine data. Thus it would seem more appropriate to talk about 

our mathematics data by first indicating that there are two major methods 

V?e have observed in mathematics instruction. Programs which are highly 

individualized and operate on a self-rpaced 'plan with uniform goals for 

all children, and programs which are enacted figr the whole class assuming •" 
" ' ' ' ' V* 



that tell children will keep pace. In some of the 'latter classes a small 
number of advanced or delayed students may be seperated in order to go . 

beyond the wholfe class goals or for remediation * and this is often done . 

* • « 
with *in<M.vidualized approaches. 

• » 

Combining data within 'these two clusters of classrooms would seem 
both more meaningful^ and interpretable in terms of either description or 

( " . " • • . f . • 

prediction^ Variations wi,thiji these clusters would allow one to investigate 

the relativ# efficacy of certain transactions or arrangements, but within 

i . 
the framework of a pedagogical context that was really operating in class- 

r m ' * 

rooms. While curricular contrasts may be helpful and appropriate on 

occasion, studies often fail because so many treatment configurations are. 
combined that patterns of relations' cannot emerge. ^ 

' A 

It is in this respect that the problems facecf by persons conducting 
quantitative research s^ithese.^ are very similar to tfcose* of the observa- 
tional researchers. HOW does one decide when it is appropriate to combine 

• ■ I 1 * 

i 

data from a variety of studies? What does it mean to be measuring the same 

~ " * i * ' ■ 

effect across studies? Light* an£ Pillemer (1962) Have suggested the 

4 * * 

necessity for including both narrative and statistical indicators of 
treatment similarities in order to interpret quantitative research syntheses. 
Giaconia and* Hedges (1981) have illustrated analyses which consider cur- * 
ricular factors in combining studies of open education. They identified 

seven features identified with the practice of open education, four of 

t > < • 

which* were deemed essential for a well-implemented open education program. 

In conducting a quantitative research synthesis every study was coded for* 

the s'^ven* features they identified. In doing this they operationally 

• i « 

defined th|a treatments that were represented in the studies the^ were 1 

•i » • 

reviewing. By examining featureg^of open education, they discovered that 



\ 
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• certain effects were strongly associated with the occurence of the program 
* i • . 

features-. In particular, all studies* which showed l^rge effect; Sizes, for 

self ^concept as an education^ outcome we£e found to* have* the four essen- 

tial faatures *they identified. TM Gfaconia and Hedges X'1981)* ex4mple s * s t 

•• * 1 

illustrates ,the actual s^eps one might take to assure treatment similarity . • 

\ - ' . " ' ' ' ' ■ . " * ' * 

in studies before looking for consist ent« q^fects across, the studies. This 

/ ' ? - u - ■ 

# step seems as necessary within observational research* studies as it does . ' • . 

* *»*»•* 
itf combining studies. - * " V 

t * "In both observational studies, and quantitative research synthesis, 

more detective work must .be done about the nature o£ the educational 

phenomena ^bein^ studied.'. Reseracheirs and writers would advance the 

state of* the art by including more in -the way of background descriptive 

•information about th6 programs #feind <pontexts which they have studied to- » 
facilitate sorting and resorting /of data. /A recognition that educational 
settings Really are multivariate in^the most, fundamental sehse seems 
essential to advance both descriptive work in education and subs^tfufent 

, analyses and rean^lyses of existing data. " * * 
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